Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] Make ImageFiltering.imfilter pipeline differentiable #21

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

[WIP] Make ImageFiltering.imfilter pipeline differentiable #21

wants to merge 4 commits into from
+149 −0

Conversation

SomTambe
Copy link
Member

@SomTambe SomTambe commented Aug 25, 2021 •
edited
Loading

This PR aims to make the ImageFiltering.imfilter function differentiable.

  • Add adjoints
  • Get the gradient of the kernel
  • Put tests

The current issue I am facing causes the gradient of the kernel to be nothing. There seems to be a place where I have incorrectly put a NoTangent(), or a @nograd. Infact upon some further tracing I found out it does go inside the rule of __imfilter_inbounds!, but never calls the pullback inside it.

julia> img = testimage("cameraman");

julia> gradf(x, k, args...) = Zygote.gradient((a, b)->imfilter(a, b, args...) |> sum |> (z->z.val), x, k)
gradf (generic function with 1 method)

julia> gradf(img, t*t') 
(Tangent{Gray{Float64}, NamedTuple{(:val,), Tuple{Float64}}}[Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,) … Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,); Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,) … Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,); … ; Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,) … Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,); Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,) … Tangent{Gray{Float64}}(val = 1.0,) Tangent{Gray{Float64}}(val = 1.0,)], nothing)

The img gradient is incorrect too, and I shall get it rectified when I write tests and figure out the kernel thing.

@codecov-commenter
Copy link

Codecov Report

Merging #21 (cf6fbf8) into main (890e927) will decrease coverage by 32.67%.
The diff coverage is 0.00%.

Impacted file tree graph

@@             Coverage Diff             @@
##             main      #21       +/-   ##
===========================================
- Coverage   88.23%   55.55%   -32.68%     
===========================================
  Files           3        4        +1     
  Lines         102      162       +60     
===========================================
  Hits           90       90               
- Misses         12       72       +60
Impacted Files Coverage Δ
src/filters/adjoints.jl 0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 890e927...cf6fbf8. Read the comment docs.

@DhairyaLGandhi
Copy link
Member

Yeah you definitely want the Tangent to wrap an array and not elements I would imagine.

DhairyaLGandhi
DhairyaLGandhi reviewed Aug 26, 2021
View reviewed changes
Copy link
Member

@DhairyaLGandhi DhairyaLGandhi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this up! Looking forward to the tests. I'll also take a look through the adjoints to see if they're correct

src/filters/adjoints.jl
z)
y = __imfilter_inbounds!(r, out, A, kern, border, R, z)
function ∇__imfilter_inbounds!(Δy)
# ∇out should not have any gradients
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it will be used further down in the computation pipeline, i think you still need to provide it the gradient so it can be accumulated properly. Else the answer would be incorrect

src/filters/adjoints.jl
y = imfilter!(out, img, kernel, border, alg)
function ∇imfilter!_try(Δy)
k = default_resource(alg_defaults(alg, out, kernel))
ret = imfilter!(k, out, img, kernel, border)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this true generically?

@johnnychen94 johnnychen94 mentioned this pull request Nov 1, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DhairyaLGandhi DhairyaLGandhi DhairyaLGandhi left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@SomTambe @codecov-commenter @DhairyaLGandhi